Method and apparatus for encoding and decoding video using inter-prediction

ABSTRACT

A coding tool is provided to improve the compression performance of inter prediction and is used at the encoder/decoder side to adjust the correction of a motion vector based on a high level syntax. In addition, a method is provided for simply performing an integer sample search step of searching for an integer offset and a fractional sample refinement step of searching for a sub-pixel offset in relation to motion vector refinement among coding tools.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to Korean Patent Application Nos.10-2019-0081926, 10-2019-0170099, and 10-2020-10-2020-0083978respectively filed on Jul. 8, 2019, Dec. 18, 2019, and Jul. 8, 2020, theentire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to encoding and decoding of a video, andmore specifically, to a coding tool configured to improve compressionperformance of inter-prediction and a method for adjusting thecorrection of motion vectors using the coding tool at theencoder/decoder side based on a high-level syntax.

BACKGROUND ART

The statements in this section merely provide background informationrelated to the present disclosure and may not constitute prior art.Since the volume of video data is greater than that of voice data orstill image data, storing or transmitting video data without processingfor compression requires a lot of hardware resources including memory.

Accordingly, in storing or transmitting video data, the video data isgenerally compressed using an encoder to be stored or transmitted. Then,a decoder receives the compressed video data, and decompresses andreproduces the video data. Compression techniques for such video includeH.264/AVC and High Efficiency Video Coding (HEVC), which improves codingefficiency over H.264/AVC by about 40%. However, the picture size,resolution, and frame rate are gradually increasing, and accordingly theamount of data to be encoded is also increasing. Accordingly, a newcompression technique having better encoding efficiency and higher imagequality than the existing compression technique is required.

In video encoding, predictive encoding is mainly used to improvecompression performance. There are intra-prediction for predicting atarget block to be encoded based on pre-reconstructed samples in acurrent picture, and inter-prediction for predicting a current blockusing a pre-reconstructed reference picture. In particular, theinter-prediction is widely used for video encoding because it exhibitssuperior compression performance compared to intra-prediction.

The present disclosure proposes a coding tool for improving thecompression performance of the existing inter-prediction.

SUMMARY

In the present disclosure, regarding a coding tool for improvingcompression performance of inter-prediction, a correction of motionvectors is adjusted using the coding tool at an encoder/decoder sidebased on a high-level syntax. It is a main object of the presentdisclosure to provide a method for simply performing an integer samplesearch operation and a fractional sample refinement operation inrelation to motion vector refinement other coding tools.

In accordance with one aspect of the present disclosure, a videodecoding method is provided for decoding a sequence of coded pictures ona block-by-block basis by a video decoding apparatus. The methodincludes: decoding at least one first high level syntax elementindicating whether each of at least one coding tool from a bitstream isallowed, the at least one coding tool including motion vectorrefinement; decoding, from the bitstream, a second high level syntaxelement corresponding to each of the at least one coding tool dependingon the at least one first high level syntax element and setting, at apicture level, whether each of the at least one coding tool is allowed;when it is identified that the motion vector refinement is allowed for acurrent picture including a target block based on the setting at thepicture level, generating a refined motion vector for the target blockby calculating a metric between two candidate blocks within a presetoffset range from initial motion vectors in bi-directional referencepictures of the target block and searching for a motion vector candidatehaving a minimum metric; and generating inter-prediction samples for thetarget block using the bi-directional reference pictures and the refinedmotion vector.

In accordance with another aspect of the present disclosure, a videodecoding apparatus is provided for decoding a sequence of coded pictureson a block-by-block basis. The apparatus includes: a decoder configuredto decode at least one first high level syntax element indicatingwhether each of at least one coding tool from a bitstream is allowed,and decode, from the bitstream, a second high level syntax elementcorresponding to each of the at least one coding tool depending on theat least one first high level syntax element and setting, at a picturelevel, whether each of the at least one coding tool is allowed, the atleast one coding tool including motion vector refinement; and apredictor configured to generate, in response to identifying that themotion vector refinement is allowed for a current picture including atarget block based on the setting at the picture level, a refined motionvector for the target block by calculating a metric between twocandidate blocks within a preset offset range from initial motionvectors in bi-directional reference pictures of the target block andsearching for a motion vector candidate having a minimum metric, whereinthe predictor generates inter-prediction samples for the target blockusing the bi-directional reference pictures and the refined motionvector.

In accordance with another aspect of the present disclosure, a videoencoding method is provided for encoding a sequence of coded pictures ona block-by-block basis by a video encoding apparatus. The methodincludes: encoding, in a sequence header, at least one first high levelsyntax element indicating whether each of at least one coding tool isallowed, the at least one coding tool including motion vectorrefinement; encoding a second high level syntax element of a picturelevel related to whether each of the at least one coding tool is alloweddepending on the at least one first high level syntax element; when itis identified that the motion vector refinement is allowed for a currentpicture including a target block based on setting of the picture level,generating a refined motion vector for the target block by calculating ametric between two candidate blocks within a preset offset range frominitial motion vectors in bi-directional reference pictures of thetarget block and searching for a motion vector candidate having aminimum metric; and generating inter-prediction samples for the targetblock using the bi-directional reference pictures and the refined motionvector.

As is apparent from the above description, according to the presentembodiment, a method is provided for simply performing an integer samplesearch operation and a fractional sample refinement operation inrelation to motion vector refinement among coding tools for improvingcompression performance of inter-prediction. Thereby, coding efficiencyof the encoder/decoder may be enhanced and accordingly complexity may bereduced.

DESCRIPTION OF DRAWINGS

FIG. 1 is an exemplary block diagram of a video encoding apparatuscapable of implementing the techniques of the present disclosure.

FIG. 2 exemplarily shows block partitioning structure using a QTBTTTstructure.

FIG. 3A-3B exemplarily show a plurality of intra-prediction modesincluding wide angle intra-prediction modes.

FIG. 4 exemplarily shows neighboring blocks around a current block.

FIG. 5 is an exemplary block diagram of a video decoding apparatuscapable of implementing the techniques of the present disclosure.

FIG. 6 is an exemplary diagram illustrating the concept of BDOFaccording to an embodiment of the present disclosure.

FIG. 7 is an exemplary diagram illustrating a method for deriving agradient for a block boundary sample in a bi-directional optical flow.

FIGS. 8A-8B are exemplary diagrams illustrating affine motion predictionaccording to an embodiment of the present disclosure.

FIG. 9 is an exemplary diagram illustrating subblock-by-subblock affinemotion prediction according to an embodiment of the present disclosure

FIG. 10 is an exemplary diagram illustrating a method for deriving mergecandidates for affine motion prediction from translation motion vectorsof neighboring blocks.

FIG. 11 is an exemplary diagram illustrating the concept of a motionvector refinement coding tool according to an embodiment of the presentdisclosure.

FIG. 12 is an exemplary diagram illustrating a method for reducingsearch complexity for the motion vector refinement coding tool accordingto an embodiment of the present disclosure.

FIG. 13 is an exemplary diagram illustrating a method for reducingsearch complexity for the motion vector refinement coding tool accordingto another embodiment of the present disclosure.

FIG. 14 is an exemplary diagram illustrating a method for reducingsearch complexity for the motion vector refinement coding tool accordingto another embodiment of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, some embodiments of the present disclosure will bedescribed in detail with reference to the accompanying drawings. Itshould be noted that, in adding reference numerals to the constituentelements in the respective drawings, like reference numerals designatelike elements, although the elements are shown in different drawings.Further, in the following description of embodiments of the presentdisclosure, a detailed description of known functions and configurationsincorporated herein will be omitted to avoid obscuring the subjectmatter of the embodiments.

FIG. 1 is an exemplary block diagram of a video encoding apparatuscapable of implementing the techniques of the present disclosure.Hereinafter, a video encoding apparatus and elements of the apparatuswill be described with reference to FIG. 1.

The video encoding apparatus includes a picture splitter 110, apredictor 120, a subtractor 130, a transformer 140, a quantizer 145, arearrangement unit 150, an entropy encoder 155, an inverse quantizer160, an inverse transformer 165, an adder 170, a loop filter unit 180,and a memory 190. Each element of the video encoding apparatus may beimplemented in hardware or software, or a combination of hardware andsoftware. The functions of the respective elements may be implemented assoftware, and a microprocessor may be implemented to execute thesoftware functions corresponding to the respective elements.

One video is composed of one or more sequences including a plurality ofpictures. Each picture is split into a plurality of regions, andencoding is performed on each region. For example, one picture is splitinto one or more tiles or/and slices. In particular, the one or moretiles may be defined as a tile group. Each tile or slice is split intoone or more coding tree units (CTUs). Each CTU is split into one or morecoding units (CUs) by a tree structure. Information applied to each CUis encoded as a syntax of the CU, and information applied to CUsincluded in one CTU in common is encoded as a syntax of the CTU. Inaddition, information applied to all blocks in one slice in common isencoded as a syntax of a slice header, and information applied to allblocks constituting one or more pictures is encoded in a pictureparameter set (PPS) or a picture header.

In addition, information referred to by one or more SPSs in common isencoded in a video parameter set (VPS). Furthermore, information which aplurality of pictures refers to in common is encoded in a sequenceparameter set (SPS). In addition, information applied to one tile ortile group in common may be encoded as a syntax of a tile or tile groupheader. The syntaxes included in the SPS, PPS, slice header, and tile ortile group header may be referred to as high-level syntaxes.

The picture splitter 110 is configured to determine the size of a codingtree unit (CTU). Information about the size of the CTU (CTU size) isencoded as a syntax of the SPS or PPS and is transmitted to the videodecoding apparatus. The picture splitter 110 splits each pictureconstituting the video into a plurality of CTUs having a predeterminedsize, and then recursively splits the CTUs using a tree structure. Inthe tree structure, a leaf node serves as a coding unit (CU), which is abasic unit of coding.

The tree structure may be a QuadTree (QT), in which a node (or parentnode) is split into four sub-nodes (or child nodes) of the same size, aBinaryTree (BT), in which a node is split into two sub-nodes, aTernaryTree (TT), in which a node is split into three sub-nodes at aratio of 1:2:1, or a structure formed by a combination of two or more ofthe QT structure, the BT structure, and the TT structure. For example, aQuadTree plus BinaryTree (QTBT) structure may be used, or a QuadTreeplus BinaryTree TernaryTree (QTBTTT) structure may be used. Here, BTTTmay be collectively referred to as a multiple-type tree (MTT).

FIG. 2 is a diagram illustrating a method for splitting a block using aQTBTTT structure. As shown in FIG. 2, a CTU may be initially split inthe QT structure. The QT splitting may be repeated until the size of thesplitting block reaches the minimum block size MinQTSize of a leaf nodeallowed in the QT. A first flag (QT_split_flag) indicating whether eachnode of the QT structure is split into four nodes of a lower layer isencoded by the entropy encoder 155 and signaled to the video decodingapparatus. When the leaf node of the QT is equal to or less than themaximum block size (MaxBTSize) of the root node allowed in the BT, itmay be further split into one or more of the BT structure or the TTstructure. The BT structure and/or the TT structure may have a pluralityof splitting directions. For example, there may be two directions,namely, a direction in which a block of a node is horizontally split anda direction in which the block is vertically split. As shown in FIG. 2,when MTT splitting is started, a second flag (mtt_split_flag) indicatingwhether nodes are split, a flag indicating a splitting direction(vertical or horizontal) in the case of splitting, and/or a flagindicating a splitting type (Binary or Ternary) are encoded by theentropy encoder 155 and signaled to the video decoding apparatus.

Alternatively, prior to encoding the first flag (QT_split_flag)indicating whether each node is split into 4 nodes of a lower layer, aCU splitting flag (split_cu_flag) indicating whether the node is splitmay be encoded. When the value of the CU split flag (split_cu_flag)indicates that splitting is not performed, the block of the node becomesa leaf node in the splitting tree structure and serves a coding unit(CU), which is a basic unit of encoding. When the value of the CU splitflag (split_cu_flag) indicates that splitting is performed, the videoencoding apparatus starts encoding the flags in the manner describedabove, starting with the first flag.

When QTBT is used as another example of a tree structure, there may betwo splitting types, which are a type of horizontally splitting a blockinto two blocks of the same size (i.e., symmetric horizontal splitting)and a type of vertically splitting a block into two blocks of the samesize (i.e., symmetric vertical splitting). A split flag (split_flag)indicating whether each node of the BT structure is split into block ofa lower layer and splitting type information indicating the splittingtype are encoded by the entropy encoder 155 and transmitted to the videodecoding apparatus. There may be an additional type of splitting a blockof a node into two asymmetric blocks. The asymmetric splitting type mayinclude a type of splitting a block into two rectangular blocks at asize ratio of 1:3, or a type of diagonally splitting a block of a node.

Further, CUs may have various sizes according to QTBT or QTBTTTsplitting of a CTU. Hereinafter, a block corresponding to a CU (i.e., aleaf node of QTBTTT) to be encoded or decoded is referred to as a“current block.” As QTBTTT splitting is employed, the shape of thecurrent block may be square or rectangular. The predictor 120 isconfigured to predict the current block to generate a prediction block.The predictor 120 includes an intra-predictor 122 and an inter-predictor124.

In general, each of the current blocks in a picture may be predictivelycoded. In addition, prediction of a current block may be performed usingan intra-prediction technique (which uses data from a picture containingthe current block) or an inter-prediction technique (which uses datafrom a picture coded before the picture containing the current block).The inter-prediction includes both unidirectional prediction andbi-directional prediction.

The intra-prediction unit 122 is configured to predict pixels in thecurrent block using pixels (reference pixels) positioned around thecurrent block in the current picture including the current block. Thereis a plurality of intra-prediction modes according to the predictiondirections. For example, as shown in FIGS. 3A-3B, the plurality ofintra-prediction modes may include two non-directional modes, whichinclude a planar mode and a DC mode, and 65 directional modes.Neighboring pixels and an equation to be used are defined differentlyfor each prediction mode.

For efficient directional prediction for a rectangular-shaped currentblock, directional modes (intra-prediction modes 67 to 80 and −1 to −14)indicated by dotted arrows in FIG. 3B may be additionally used. Thesemodes may be referred to as “wide angle intra-prediction modes.” In FIG.3B, arrows indicate corresponding reference samples used for prediction,not indicating prediction directions. The prediction direction isopposite to the direction indicated by an arrow. A wide-angle intraprediction mode is a mode in which prediction is performed in adirection opposite to a specific directional mode without additional bittransmission when the current block has a rectangular shape.

In particular, among the wide angle intra-prediction modes, some wideangle intra-prediction modes available for the current block may bedetermined based on a ratio of the width and height of the rectangularcurrent block. For example, wide angle intra-prediction modes with anangle less than 45 degrees (intra prediction modes 67 to 80) may be usedwhen the current block has a rectangular shape with a height less thanthe width thereof. Wide angle intra-prediction modes with an anglegreater than −135 degrees (intra-prediction modes −1 to −14) may be usedwhen the current block has a rectangular shape with a width greater thanthe height thereof.

The intra-predictor 122 may be configured to determine anintra-prediction mode to be used in encoding the current block. In someexamples, the intra-predictor 122 may be configured to encode thecurrent block using several intra-prediction modes and select anappropriate intra-prediction mode to use from the tested modes. Forexample, the intra-predictor 122 may be configured to calculate ratedistortion values using rate-distortion analysis of several testedintra-prediction modes, and select an intra-prediction mode that has thebest rate distortion characteristics among the tested modes.

The intra-predictor 122 is configured to select one intra-predictionmode from among the plurality of intra-prediction modes, and predict thecurrent block using neighboring pixels (reference pixel) and an equationdetermined according to the selected intra-prediction mode. Informationabout the selected intra-prediction mode is encoded by the entropyencoder 155 and transmitted to the video decoding apparatus.

The inter-predictor 124 is configured to generate a prediction block forthe current block through motion compensation. The inter-predictor 124is configured to search for a block most similar to the current block ina reference picture which has been encoded and decoded earlier than thecurrent picture, and generate a prediction block for the current blockusing the searched block. Then, the inter-predictor is configured togenerate a motion vector (MV) corresponding to a displacement betweenthe current block in the current picture and the prediction block in thereference picture. In general, motion estimation is performed on a lumacomponent, and a motion vector calculated based on the luma component isused for both the luma component and the chroma component. The motioninformation including information about the reference picture andinformation about the motion vector used to predict the current block isencoded by the entropy encoder 155 and transmitted to the video decodingapparatus.

The inter-predictor 124 may be configured to perform interpolation on areference picture or a reference block to increase prediction accuracy.In other words, subsamples between two consecutive integer samples areinterpolated by applying filter coefficients to a plurality ofconsecutive integer samples including the two integer samples. When theoperation of searching for a block most similar to the current block isperformed on the interpolated reference picture, the motion vector maybe expressed at a precision level of fractional sample unit, not aprecision level of integer sample unit.

The precision or resolution of the motion vector may be set differentlyfor each target region to be encoded, for example, each unit such as aslice, tile, CTU, or CU. When such an adaptive motion vector resolution(AMVR) is applied, information about motion vector resolution to beapplied to each target region should be signaled for each target region.For example, when the target region is a CU, information about themotion vector resolution applied to each CU is signaled. The informationabout the motion vector resolution may be information indicating theprecision of a motion vector difference which will be described later.

The inter-predictor 124 may be configured to perform inter-predictionusing bi-prediction. In bi-directional prediction, the inter-predictor124 uses two reference pictures and two motion vectors representingblock positions most similar to the current block in the respectivereference pictures. The inter-predictor 124 is configured to select afirst reference picture and a second reference picture from referencepicture list 0 (RefPicList0) and reference picture list 1 (RefPicList1),respectively, search for blocks similar to the current block in therespective reference pictures, and generate a first reference block anda second reference block. Then, the inter-predictor 124 is configured togenerate a prediction block for the current block by averaging orweighting the first reference block and the second reference block. Theinter-predictor 124 is then configured to transfer motion informationincluding information about the two reference pictures and the twomotion vectors used to predict the current block to the encoder 150. Inparticular, RefPicList0 may be composed of pictures preceding thecurrent picture in display order among the reconstructed pictures, andRefPicList1 may be composed of pictures following the current picture indisplay order among the reconstructed pictures. However, embodiments arenot limited thereto. Pre-reconstructed pictures following the currentpicture in display order may be further included in RefPicList0, andconversely, and pre-reconstructed pictures preceding the current picturemay be further included in RefPicList1.

Various methods may be used to minimize the number of bits required toencode the motion information. For example, when the reference pictureand motion vector of the current block are the same as the referencepicture and motion vector of a neighboring block, the motion informationabout the current block may be transmitted to the video decodingapparatus by encoding information for identifying the neighboring block.This method is called a “merge mode.”

In the merge mode, the inter-predictor 124 is configured to select apredetermined number of merge candidate blocks (hereinafter referred toas “merge candidates”) from among the neighboring blocks of the currentblock. As illustrated in FIG. 4, all or part of a left block L, an aboveblock A, an above right block AR, a bottom left block BL, and an aboveleft block AL, which are adjacent to the current block in the currentpicture, may be used as neighboring blocks for deriving mergecandidates. In addition, a block located within a reference picture(which may be the same as or different from the reference picture usedto predict the current block) other than the current picture in whichthe current block is located may be used as a merge candidate. Forexample, a co-located block which is at the same position as the currentblock or blocks adjacent to the co-located block in the referencepicture may be additionally used as merge candidates.

The inter-predictor 124 configures a merge list including apredetermined number of merge candidates using such neighboring blocks.The inter-predictor 124 is configured to select a merge candidate to beused as the motion information about the current block from among themerge candidates included in the merge list, and generate merge indexinformation for identifying the selected candidates. The generated mergeindex information is encoded by the encoder 155 and transmitted to thevideo decoding apparatus.

Another method of encoding the motion information is an advanced motionvector prediction (AMVP) mode. In the AMVP mode, the inter-predictor 124is configured to derive predicted motion vector candidates for themotion vector of the current block by using neighboring blocks of thecurrent block. All or part of the left block L, the above block A, theabove right block AR, the bottom left block BL, and the above left blockAL, which are adjacent to the current block in the current picture inFIG. 2, may be used as the neighboring blocks used to derive thepredicted motion vector candidates.

In addition, a block positioned within a reference picture (which may bethe same as or different from the reference picture used to predict thecurrent block) other than the current picture including the currentblock may be used as the neighboring blocks used to derive the predictedmotion vector candidates. For example, a co-located block which is atthe same position as the current block or blocks adjacent to thecollocated block in the reference picture may be used. Theinter-predictor 124 is configured to derive predicted motion vectorcandidates using the motion vectors of the neighboring blocks, anddetermine a predicted motion vector for the motion vector of the currentblock using the predicted motion vector candidates. Then, a motionvector difference is calculated by subtracting the predicted motionvector from the motion vector of the current block.

The predicted motion vector may be obtained by applying a predefinedfunction (e.g., a function for calculating a median, an average, or thelike) to the predicted motion vector candidates. In particular, thevideo decoding apparatus also knows the predefined function. Since theneighboring blocks used to derive the predicted motion vector candidateshave already been encoded and decoded, the video decoding apparatusalready knows the motion vectors of the neighboring blocks as well.Accordingly, the video encoding apparatus does not need to encodeinformation for identifying the predicted motion vector candidates.Therefore, in this case, the information about the motion vectordifference and the information about the reference picture used topredict the current block are encoded.

The predicted motion vector may be determined by selecting any one ofthe predicted motion vector candidates. In particular, information foridentifying the selected predicted motion vector candidate is furtherencoded along with the information about the motion vector differenceand the information about the reference picture which are to be used topredict the current block. The subtractor 130 is configured to subtractthe prediction block generated by the intra-predictor 122 or theinter-predictor 124 from the current block to generate a residual block.

The transformer 140 is configured to transform the residual signal inthe residual block having pixel values in the spatial domain intotransform coefficients in the frequency domain. The transformer 140 maybe configured to transform residual signals in the residual block usingthe entire size of the residual block as a transformation unit.Alternatively, the transformer 140 may be configured to split theresidual block into a plurality of subblocks, and perform thetransformation using the subblock as a transform unit. Alternatively,the residual signals may be transformed by dividing the block into twosubblocks, which are a transform region and a non-transform region, andusing only the transform region subblock as a transform unit.

Particularly, the transform region subblock may be one of tworectangular blocks having a size ratio of 1:1 based on the horizontalaxis (or vertical axis). In this case, a flag (cu_sbt_flag) indicatingthat only subblocks have been transformed, directionality(vertical/horizontal) information (cu_sbt_horizontal_flag) and/orposition information (cu_sbt_pos_flag) are encoded by the entropyencoder 155 and signaled to the video decoding apparatus. In addition,the size of the transform region subblock may have a size ratio of 1:3based on the horizontal axis (or vertical axis). In particular, a flag(cu_sbt_quad_flag) distinguishing the corresponding splitting isadditionally encoded by the entropy encoder 155 and signaled to thevideo decoding apparatus.

The transformer 140 may be configured to transform the residual block inthe horizontal direction and the vertical direction individually. Fortransformation, various types of transform functions or transformmatrices may be used. For example, pairs of transform functions forhorizontal transformation and vertical transformation may be defined asa multiple transform set (MTS). The transformer 140 may be configured toselect one pair of transform functions having the best transformationefficiency in the MTS, and transform the residual block in thehorizontal and vertical directions, respectively. The information(mts_idx) on the transform function pair selected in the MTS is encodedby the entropy encoder 155 and signaled to the video decoding apparatus.

The quantizer 145 is configured to quantize transform coefficientsoutput from the transformer 140 using quantization parameters, andoutput the quantized transform coefficients to the entropy encoder 155.For some blocks or frames, the quantizer 145 may be configured todirectly quantize a related residual block without transformation. Thequantizer 145 may be configured to apply different quantizationcoefficients (scaling values) according to the positions of thetransform coefficients in a transform block. A matrix of quantizedcoefficients applied to the two-dimensionally arranged quantizedtransform coefficients may be encoded and signaled to the video decodingapparatus.

The rearrangement unit 150 may be configured to re-sort the coefficientvalues for the quantized residual value. The rearrangement unit 150 maybe configured to change the 2-dimensional array of coefficients into a1-dimensional coefficient sequence through coefficient scanning. Forexample, the rearrangement unit 150 may be configured to scancoefficients from a direct current (DC) coefficient to a coefficient ina high frequency region using a zig-zag scan or a diagonal scan tooutput a 1-dimensional coefficient sequence. Depending on the size ofthe transformation unit and the intra-prediction mode, a vertical scan,in which a two-dimensional array of coefficients is scanned in a columndirection, or a horizontal scan, in which two-dimensional block-shapedcoefficients are scanned in a row direction, may be used instead of thezig-zag scan. In other words, a scan mode to be used may be determinedamong the zig-zag scan, the diagonal scan, the vertical scan and thehorizontal scan according to the size of the transformation unit and theintra-prediction mode.

The entropy encoder 155 is configured to encode the one-dimensionalquantized transform coefficients output from the rearrangement unit 150using uses various encoding techniques such as Context-based AdaptiveBinary Arithmetic Code (CABAC) and exponential Golomb, to generate abitstream. The entropy encoder 155 is configured to encode informationsuch as a CTU size, a CU split flag, a QT split flag, an MTT splittingtype, and an MTT splitting direction, which are associated with blocksplitting, such that the video decoding apparatus may split the block inthe same manner as in the video encoding apparatus.

In addition, the entropy encoder 155 is configured to encode informationabout a prediction type indicating whether the current block is encodedby intra-prediction or inter-prediction, and encode intra-predictioninformation (i.e., information about an intra-prediction mode) orinter-prediction information (a merge index for the merge mode,information about a reference picture index and a motion vectordifference for the AMVP mode) according to the prediction type. Theentropy encoder 155 is also configured to encode information related toquantization, that is, information about quantization parameters andinformation about a quantization matrix.

The inverse quantizer 160 is configured to inversely quantize thequantized transform coefficients output from the quantizer 145 togenerate transform coefficients. The inverse transformer 165 isconfigured to transform the transform coefficients output from theinverse quantizer 160 from the frequency domain to the spatial domainand reconstructs the residual block. The adder 170 is configured to addthe reconstructed residual block to the prediction block generated bythe predictor 120 to reconstruct the current block. The pixels in thereconstructed current block are used as reference pixels in performingintra-prediction of a next block.

The loop filter unit 180 is configured to filter the reconstructedpixels to reduce blocking artifacts, ringing artifacts, and blurringartifacts generated due to block-based prediction andtransformation/quantization. The loop filter unit 180 may include atleast of a deblocking filter 182, a sample adaptive offset (SAO) filter184, and an adaptive loop filter (ALF) 186. The deblocking filter 182 isconfigured to filter the boundary between the reconstructed blocks toremove blocking artifacts caused by block-by-block coding/decoding, andthe SAO filter 184 is configured to perform additional filtering on thedeblocking-filtered video.

In particular, the SAO filter 184 is a filter used to compensate for adifference between a reconstructed sample and an original sample causedby lossy coding, and performs filtering in a manner that adding acorresponding offset to each reconstructed sample. The ALF 186 isconfigured to perform filtering on a target sample to be filtered byapplying filter coefficients to the target sample and neighboringsamples of the target sample. The ALF 186 may be configured to dividethe samples included in a picture into predetermined groups, and thendetermine one filter to be applied to a corresponding group todifferentially perform filtering on each group. Information about filtercoefficients to be used for the ALF may be encoded and signaled to thevideo decoding apparatus. The reconstructed blocks filtered by the loopfilter unit 180 are stored in the memory 190. Once all blocks in onepicture are reconstructed, the reconstructed picture may be used as areference picture for inter-prediction of blocks in a picture to beencoded next.

FIG. 5 is an exemplary functional block diagram of a video decodingapparatus capable of implementing the techniques of the presentdisclosure. Hereinafter, the video decoding apparatus and elements ofthe apparatus will be described with reference to FIG. 5.

The video decoding apparatus may include an entropy decoder 510, arearrangement unit 515, an inverse quantizer 520, an inverse transformer530, a predictor 540, an adder 550, a loop filter unit 560, and a memory570. Similar to the video encoding apparatus of FIG. 1, each element ofthe video decoding apparatus may be implemented in hardware, software,or a combination of hardware and software. Further, the function of eachelement may be implemented in software, and the microprocessor may beimplemented to execute the function of software corresponding to eachelement.

The entropy decoder 510 is configured to determine a current block to bedecoded by decoding a bitstream generated by the video encodingapparatus and extracting information related to block splitting, andextract prediction information and information about a residual signal,and the like required to reconstruct the current block. The entropydecoder 510 is configured to extract information about the CTU size fromthe sequence parameter set (SPS) or the picture parameter set (PPS),determine the size of the CTU, and split a picture into CTUs of thedetermined size. Then, the decoder is configured to determine the CTU asthe uppermost layer, that is, the root node of a tree structure, andextract splitting information about the CTU to split the CTU using thetree structure.

For example, when the CTU is split using a QTBTTT structure, a firstflag (QT_split_flag) related to splitting of the QT is extracted tosplit each node into four nodes of a sub-layer. For a node correspondingto the leaf node of the QT, the second flag (MTT_split_flag) andinformation about a splitting direction (vertical/horizontal) and/or asplitting type (binary/ternary) related to the splitting of the MTT areextracted to split the corresponding leaf node in the MTT structure.Thereby, each node below the leaf node of QT is recursively split in aBT or TT structure.

As another example, when a CTU is split using the QTBTTT structure, a CUsplit flag (split_cu_flag) indicating whether to split a CU may beextracted. When the corresponding block is split, the first flag(QT_split_flag) may be extracted. In the splitting operation, zero ormore recursive MTT splitting may occur for each node after zero or morerecursive QT splitting. For example, the CTU may directly undergo MTTsplitting without the QT splitting, or undergo only QT splittingmultiple times. As another example, when the CTU is split using the QTBTstructure, the first flag (QT_split_flag) related to QT splitting isextracted, and each node is split into four nodes of a lower layer.Then, a split flag (split_flag) indicating whether a node correspondingto a leaf node of QT is further split in the BT and the splittingdirection information are extracted.

Once the current block to be decoded is determined through splitting inthe tree structure, the entropy decoder 510 is configured to extractinformation about a prediction type indicating whether the current blockis intra-predicted or inter-predicted. When the prediction typeinformation indicates intra-prediction, the entropy decoder 510 isconfigured to extract a syntax element for the intra-predictioninformation (intra-prediction mode) for the current block. When theprediction type information indicates inter-prediction, the entropydecoder 510 is configured to extract a syntax element for theinter-prediction information, that is, information indicating a motionvector and a reference picture referred to by the motion vector.

The entropy decoder 510 is also configured to extract information aboutquantized transform coefficients of the current block as informationrelated to quantization and information about residual signals. Therearrangement unit 515 may be configured to change the sequence of theone-dimensional quantized transform coefficients entropy-decoded by theentropy decoder 510 to a 2-dimensional coefficient array (i.e., block)in a reverse order of the coefficient scanning performed by the videoencoding apparatus.

The inverse quantizer 520 is configured to inversely quantize thequantized transform coefficients, and inversely quantizes transformcoefficients quantized using the quantization parameter. The inversequantizer 520 may be configured to apply different quantizationcoefficients (scaling values) to the quantized transform coefficientsarranged in two dimensions. The inverse quantizer 520 may be configuredto perform inverse quantization by applying a matrix of quantizationcoefficients (scaling values) from the video encoding apparatus to atwo-dimensional array of quantized transform coefficients. The inversetransformer 530 is configured to inversely transform the inverselyquantized transform coefficients from the frequency domain to thespatial domain to reconstruct residual signals, thereby generating aresidual block for the current block.

In addition, when the inverse transformer 530 inversely transforms onlya partial region (subblock) of the transform block, it extracts a flag(cu_sbt_flag) indicating that only the subblock of the transform blockhas been transformed, the directionality (vertical/horizontal)information about the subblock (cu_sbt_horizontal_flag), and/or positioninformation about the subblock (cu_sbt_pos_flag), and inverselytransforms the transform coefficients of the subblock from the frequencydomain to the spatial domain. Then, it reconstructs residual signals,and fills a region that is not inversely transformed with the value of“0” as a residual block, thereby generating the final residual block forthe current block.

In addition, when the MTS is applied, the inverse transformer 530 isconfigured to determine transform functions or transform matrices to beapplied in the horizontal and vertical directions, respectively, usingthe MTS information (mts_idx) signaled from the video encodingapparatus, and use the determined transform functions to inverselytransform the transform coefficients in the transform block in thehorizontal and vertical directions. The predictor 540 may include anintra-predictor 542 and an inter-predictor 544. The intra-predictor 542is activated when the prediction type of the current block isintra-prediction, and the inter-predictor 544 is activated when theprediction type of the current block is inter-prediction.

The intra-predictor 542 is configured to determine an intra-predictionmode of the current block among a plurality of intra-prediction modesbased on the syntax element for the intra-prediction mode extracted fromthe entropy decoder 510, and predict the current block using thereference pixels around the current block according to theintra-prediction mode. The inter-predictor 544 is configured todetermine a motion vector of the current block and a reference picturereferred to by the motion vector using the syntax element for theintra-prediction mode extracted from the entropy decoder 510, andpredict the current block based on the motion vector and the referencepicture.

The adder 550 is configured to reconstruct the current block by addingthe residual block output from the inverse transformer and theprediction block output from the inter-predictor or the intra-predictor.The pixels in the reconstructed current block are used as referencepixels in intra-predicting a block to be decoded next. The loop filterunit 560 may include a deblocking filter 562 and an SAO filter 564. Thedeblocking filter 562 deblocking-filters the boundary between thereconstructed blocks to remove blocking artifacts caused byblock-by-block decoding.

The SAO filter 564 is configured to perform additional filtering on thereconstructed block after deblocking filtering to compensate for adifference between the reconstructed pixel and the original pixel causedby lossy coding. The ALF 566 is configured to perform filtering on atarget sample to be filtered by applying filter coefficients to thetarget sample and neighboring samples of the target sample. The ALF 566may be configured to divide the samples in a picture into predeterminedgroups, and then determine one filter to be applied to a correspondinggroup to differentially perform filtering on each group. The filtercoefficient of the ALF is determined based on the information about thefilter coefficient decoded from the bitstream. The reconstructed blockfiltered by the loop filter unit 560 is stored in the memory 570. Whenall blocks in one picture are reconstructed, the reconstructed pictureis used as a reference picture for inter-prediction of blocks in apicture to be encoded next.

The present embodiment relates to encoding and decoding of a video asdescribed above. More specifically, regarding a coding tool thatimproves the compression performance of inter-prediction, theencoder/decoder adjusts the correction of motion vectors using thecoding tool based on a high-level syntax. In addition, the presentembodiment provides a method for simply performing an integer samplesearch operation of searching for an integer offset and a fractionalsample refinement operation of searching for a subpixel offset, inrelation to motion vector refinement among the coding tools.

The following embodiment relates to coding tools for improvingcompression performance of inter-prediction, which may be operated bythe inter-predictor 124 of the video encoding apparatus and theinter-predictor 544 of the video decoding apparatus. As used herein, theterm “target block” may have the same meaning as the current block orcoding unit (CU) described above, or may mean a partial region of a CU.

I. Bi-Directional Optical Flow (BDOF)

Bi-directional optical flow is a technique for additionally compensatingfor motions of samples predicted using bi-directional motion prediction,on the assumption that samples or objects constituting a video moves ata constant speed and that there is little change in sample values. FIG.6 is an exemplary diagram illustrating the concept of BDOF according toan embodiment of the present disclosure.

Suppose that bi-directional motion vectors MV0 and MV1 pointing tocorresponding regions (i.e., reference blocks) most similar to theto-be-encoded target block in the current picture have been determinedin the reference pictures Ref₀ and Ref₁ by (normal) bi-directionalmotion prediction for the target block. The two motion vectors havevalues representing the motion of the entire target block.

In the example of FIG. 6, P₀ is a sample in the reference picture Ref₀indicated by the motion vector MV₀ and corresponding to sample P in thetarget block, and P₁ is a sample in the reference picture Ref₁ indicatedby the motion vector MV₁ and corresponding to sample P in the targetblock. Further, suppose that motion for sample P in FIG. 6 is slightlydifferent from the overall motion of the target block. For example, whenan object located at sample A in Ref₀ of FIG. 6 moves to sample B inRef₁ via sample P in the target block of the current picture, sample Aand sample B may have values quite similar to each other. Accordingly,in this case, a point in Ref₀ most similar to sample P in the targetblock is not P₀ indicated by the motion vector MV₀, but sample A whichhas been shifted from P₀ by a predetermined displacement vector(v_(x)t₀, v_(y)t₁). A point in Ref₁ most similar to sample P in thetarget block is not P₁ indicated by the motion vector MV₁, but sample Bwhich has been shifted from P₁ by a predetermined displacement vector(−v_(x)t₀, −v_(y)t₁). In particular, t₀ and t₁ denote time-axisdistances of Ref₀ and Ref₁ with respect to the current picture,respectively, and are calculated based on picture order count (POC).Hereinafter, (v_(x), v_(y)) is referred to as an “optical flow.”

In predicting the value of sample P of the current block in the currentpicture, two reference samples A and B enables more accurate predictionthan reference samples P₀ and P₁ indicated by the bi-directional motionvectors MV₀ and MV₁. I⁽⁰⁾(i,j) denotes a value of a sample in thereference picture Ref₀ indicated by the motion vector MV₀ ascorresponding to the sample (i, j) in the target block, and I⁽¹⁾(i,j)denotes a value of a sample in the reference picture Ref₁ indicated bythe motion vector MV₁ as corresponding to the sample (i,j) in the targetblock.

The value of sample A in the reference picture Ref₀ indicated by a BDOFmotion vector (v_(x), v_(y)) as corresponding to the sample in thetarget block may be defined as I⁽⁰⁾(i+v_(x)t₀, j+v_(y)t₀), and the valueof sample B in the reference picture Ref₁ may be defined asI⁽¹⁾(i−v_(x)t₁, j−v_(y)t₁). In particular, when linear approximation isperformed using only the first-order term of the Taylor series, A and Bmay be expressed as Equation 1.

$\begin{matrix}{\begin{matrix}{A = {I^{(0)}\left( {{i + {v_{x}\tau_{0}}},{j + {v_{y}\tau_{0}}}} \right)}} \\{\approx {{I^{(0)}\left( {i,j} \right)} + {v_{x}\tau_{0}{I_{x}^{(0)}\left( {i,j} \right)}} + {v_{y}\tau_{0}{I_{y}^{(0)}\left( {i,j} \right)}}}}\end{matrix}\begin{matrix}{B = {I^{(1)}\left( {{i - {v_{x}\tau_{1}}},{j - {v_{y}\tau_{1}}}} \right)}} \\{\approx {{I^{(1)}\left( {i,j} \right)} - {v_{x}\tau_{1}{I_{x}^{(1)}\left( {i,j} \right)}} - {v_{y}\tau_{1}{I_{y}^{(1)}\left( {i,j} \right)}}}}\end{matrix}} & {{Equation}1}\end{matrix}$

wherein, I_(x) ^((k)) and I_(y) ^((k)) (k=0, 1) are gradient values inthe horizontal and vertical directions at position (i, j) of Ref₀ andRef₁. As described above, t₀ and t₁ denote time-axis distances of Ref₀and Ref₁ with respect to the current picture, respectively, and arerepresented as t₀=POC(current)−POC(Ref₀) and t₁=POC(Ref₁)−POC(current),based on POC.

The bi-directional optical flow (v_(x), v_(y)) of each sample in a blockis determined as a solution that minimizes Δ, which is defined as adifference between sample A and sample B. Δ may be defined by Equation 2using the linear approximation of A and B derived from Equation 1.

$\begin{matrix}\begin{matrix}{\Delta = {A - B}} \\{= {\left( {I^{(0)} - I^{(1)}} \right) + {v_{x}\left( {{\tau_{0}I_{x}^{(0)}} + {\tau_{1}I_{x}^{(1)}}} \right)} + {v_{y}\left( {{\tau_{0}I_{y}^{(0)}} + {\tau_{1}I_{y}^{(1)}}} \right)}}}\end{matrix} & {{Equation}2}\end{matrix}$

For simplicity, the sample position (i, j) is omitted from each term ofEquation 2.

To implement more robust optical flow estimation, it is assumed that themotion is locally consistent with neighboring samples. the BDOF motionvector for a sample (i, j) that is currently to be predicted iscalculated considering the differences Δ in Equation 2 for all samples(i′, j′) present in a mask Q of a certain size centered on the sample(i, j). As shown in Equation 3, the optical flow for the current sample(i, j) may be determined as a vector that minimizes the sum of squaresof the differences Δ[i′, j′] obtained for the respective samples in themask Ω, as shown in Equation 3.

$\begin{matrix}{{\Phi\left( {v_{x},v_{y}} \right)} = \ {\sum\limits_{{\lbrack{i^{\prime},j^{\prime}}\rbrack} \in \Omega}{\Delta^{2}\left\lbrack {i^{\prime},j^{\prime}} \right\rbrack}}} & {{Equation}3}\end{matrix}$

The bi-directional optical flow of the present embodiment may be appliedin the case where one of two reference pictures used for bi-directionalprediction precedes the current picture in display order, the other onefollows the current picture, and the distances from the two referencepictures to the current picture are equal to each other, that is, thedifferences in picture order count (POC) between each reference pictureand the current picture are equal to each other. Therefore, t₀ and t₁may be excluded.

In addition, the bi-directional optical flow of the present embodimentmay be applied only to luma components. The bi-directional optical flowof the present embodiment is performed on a subblock basis, not pixelbasis, for the target block to which bi-directional prediction isapplied. In the following description, the subblock, which may havevarious sizes of 2×2, 4×4, and 8×8, is assumed to have a size of 4×4 forsimplicity.

Before performing the optical flow, the inter-predictor 124 of the videoencoding apparatus is configured to generate two reference blocks forthe target block using the fore-mentioned bi-directional prediction. Thefirst reference block of the two reference blocks represents a blockcomposed of predicted samples generated from the reference picture Ref₀using the first motion vector MV₀ of the target block, and the secondreference block represents a block composed of predicted samplesgenerated from the reference picture Ref₁ using the motion vector MV₁.

The inter-predictor 124 is configured to calculate an optical flow(v_(x), v_(y)) for each of 4×4 subblocks constituting the target block,using horizontal and vertical gradient values of predicted samples inthe first reference block and the second reference block. The opticalflow (v_(x), v_(y)) is determined such that the difference between thepredicted samples from the reference picture Ref₀ and the predictedsamples from the reference picture Ref₁ is minimized. Theinter-predictor 124 is configured to derive a sample offset formodifying bi-directional predicted samples for the 4×4 subblock, byusing (v_(x), v_(y)) calculated for the 4×4 subblock and gradients ofthe predicted samples in the 4×4 subblock.

As shown in Equation 4, the inter-predictor 124 is configured tocalculate the horizontal and vertical gradients of sample values atposition (i, j).

$\begin{matrix}{{{\frac{\partial I^{(k)}}{\partial x}\left( {i,j} \right)} = {\left( {{I^{(k)}\left( {{i + 1},j} \right)} - {I^{(k)}\left( {{i - 1},j} \right)}} \right) \gg {{shift}1}}}{{\frac{\partial I^{(k)}}{\partial y}\left( {i,j} \right)} = {\left( {{I^{(k)}\left( {i,{j + 1}} \right)} - {I^{(k)}\left( {i,{j - 1}} \right)}} \right) \gg {{shift}1}}}} & {{Equation}4}\end{matrix}$

wherein, k is 0 or 1, and I⁽⁰⁾(i,j) and I⁽¹⁾(i,j) denote sample valuesat position (i,j) in the first reference block and the second referenceblock, respectively. Additionally, shift1 is a value derived frombit-depth for a luma component, for example shift1=max(6, bitDepth−6).

To derive a gradient of a sample located at the boundary of eachreference block, samples outside the boundary of the first referenceblock and the second reference block are needed. Accordingly, as shownin FIG. 6, each reference block is extended by one column to the leftand right and by one row to the upper side and lower side. To reduce theamount of computation, each sample in the extended portion may be paddedwith a sample or integer sample at the nearest position in the referenceblock. Additionally, gradients at a sample position outside the boundaryof each reference block may be padded with gradients corresponding tosamples at the nearest position.

The inter-predictor 124 is configured to calculate S1, S2, S3, S5, andS6 corresponding to auto-correlation and cross-correlation of gradientsas shown in Equation 5, using the horizontal and vertical gradients in a6×6 window covering a 4×4 subblock as shown in FIG. 7.

$\begin{matrix}\begin{matrix}{{S_{1} = {\sum_{{({i,j})} \in \Omega}{\psi_{x}{\left( {i,j} \right) \cdot {\psi_{x}\left( {i,j} \right)}}}}},} & {S_{3} = {\sum_{{({i,j})} \in \Omega}{\theta{\left( {i,j} \right) \cdot \psi_{x}}\left( {i,j} \right)}}} \\ & {S_{2} = {\sum\limits_{{({i,j})} \in \Omega}{\psi_{x}{\left( {\overset{´}{\iota},j} \right) \cdot \psi_{y}}\left( {i,j} \right)}}} \\{S_{5} = {\sum_{{({i,j})} \in \Omega}{\psi_{y}{\left( {i,j} \right) \cdot \psi_{y}}\left( {i,j} \right)}}} & {S_{6} = {\sum_{{({i,j})} \in \Omega}{\theta{\left( {i,j} \right) \cdot \psi_{y}}\left( {i,j} \right)}}}\end{matrix} & {{Equation}5}\end{matrix}$

As described above, Ω denotes a mask of a certain size. In addition, asshown in Equation 6 below, Ψ_(x)(i,j) denotes the sum of the horizontalgradient values at position (i,j) in the first reference block and thesecond reference block, Ψ_(y)(i,j) denotes the sum of the verticalgradient values at position (i,j) position in the first reference blockand the second reference block, and θ(i,j) denotes a difference betweenthe sample value at position (i,j) in the second reference block and thesample value at position (i,j) in the first reference block.

$\begin{matrix}{{{\psi_{x}\left( {i,j} \right)} = {\left( {{\frac{\partial I^{(1)}}{\partial x}\left( {i,j} \right)} + {\frac{\partial I^{(0)}}{\partial x}\left( {i,\ j} \right)}} \right) \gg n_{a}}}{{\psi_{y}\left( {i,\ j} \right)} = {\left( {{\frac{\partial I^{(1)}}{\partial y}\left( {i,j} \right)} + {\frac{\left. {\partial I^{(0)}} \right)}{\partial y}\left( {i,j} \right)}} \right) \gg n_{a}}}{{\theta\left( {i,j} \right)} = {\left( {{I^{(1)}\left( {i,j} \right)} \gg n_{b}} \right) - \left( {{I^{(0)}\left( {i,j} \right)} \gg n_{b}} \right)}}} & {{Equation}6}\end{matrix}$

wherein, n_(a) and n_(b) are values derived from the bit-depth, and havevalues of min(1, bitDepth−11) and min(4, bitDepth−8).

The inter-predictor 124 is configured to calculate the optical flows(v_(x), v_(y)) for the 4×4 subblock based on S1, S2, S3, S5, and S6, asshown in Equation 7.

$\begin{matrix}{{v_{x} = {S_{1} > {{0?{clip}}3\left( {{- {th}_{BIO}^{\prime}},{th}_{BIO}^{\prime},{- \left( {\left( {S_{3} \cdot 2^{n_{b} - n_{a}}} \right) \gg \left\lfloor {\log_{2}S_{1}} \right\rfloor} \right)}} \right):0}}}{v_{y} = {S_{5} > {{0?{clip}}3\left( {{- {th}_{BIO}^{\prime}},{th}_{BIO}^{\prime},{- \left( {\left( {{S_{6} \cdot 2^{n_{b} - n_{a}}} - {\left( {\left( {v_{x}S_{2,m}} \right) \ll {n_{S_{2}} + {v_{x}S_{2,s}}}} \right)/2}} \right) \gg \left\lfloor {\log_{2}S_{5}} \right\rfloor} \right)}} \right):0}}}{{wherein},{S_{2,m} = {{S_{2} \gg {n_{S_{2_{,}}}S_{2,s}}} = {{S_{2}\&}\left( {2^{n_{S_{2}}} - 1} \right)}}},{{{and}{th}_{BIO}^{\prime}} = {2^{\max({5,{{BD} - 7}})}.}}}} & {{Equation}7}\end{matrix}$

└⋅┘ is a floor function, and n_(S) ₂ =12.

The sample offset for modifying the predicted sample at each sampleposition (x, y) in the 4×4 subblock in the target block may becalculated using the optical flow calculated for the 4×4 subblock andgradient values at the sample position (x, y), as shown in Equation 8.

$\begin{matrix}\left. \left. {{b\left( {x,y} \right)} = {{{rnd}\left( {v_{x}\left( {\frac{\partial{I^{(1)}\left( {x,y} \right)}}{\partial x} - {\frac{\partial{I^{(0)}\left( {x,y} \right)}}{\partial x}/2}} \right)} \right)} + {r{{nd}\left( {{v_{y}\left( {\frac{\partial{I^{(1)}\left( {x,y} \right)}}{\partial x} - \frac{\partial{I^{(0)}\left( {x,y} \right)}}{\partial x}} \right)}/2} \right)}}}} \right) \right| & {{Equation}8}\end{matrix}$

wherein, rnd( ) denotes a round-off operation.

The inter-predictor 124 is configured to generate a final predictedsample pred(x, y) using the sample offset b(x, y) at position (x, y),and predicted samples I⁽⁰⁾(x,y) and I⁽¹⁾(x,y) in the first referenceblock and the second reference block, as shown in Equation 9.

pred(x,y)=(I ⁽⁰⁾(x,y)+I ⁽¹⁾(x,y)+b(x,y)+o _(offset))>>shift  Equation 9

wherein, shift is Max(3, 15-bitDepth), and O_(offset) is a value for arounding-off operation and is half the shift.

As described above, the bi-directional optical flow technique usesvalues of samples predicted using motion information (two motion vectorsand two reference pictures) used for bi-directional prediction.Therefore, the inter-predictor 544 of the video decoding apparatus mayalso perform the bi-directional optical flow in the same manner as thevideo encoding apparatus, using the motion information (motion vectors,reference pictures) for bi-directional prediction received from thevideo encoding apparatus. Accordingly, it is not required that the videoencoding apparatus signals additional information for the bi-directionaloptical flow process to the video decoding apparatus.

II. Affine Motion Prediction

The inter-prediction described above is motion prediction reflecting atranslation motion model. In other words, it is a technique forpredicting motion in the horizontal direction (x-axis direction) and thevertical direction (y-axis direction). However, in practice, there maybe various types of motions such as rotation, zoom-in, or zoom-out inaddition to the translational motion. One aspect of the presentembodiment provides affine motion prediction capable of reflecting suchvarious types of motions.

FIGS. 8A-8B are exemplary diagrams illustrating affine motion predictionaccording to an embodiment of the present disclosure.

There may be two types of models for affine motion prediction. One is amodel using the motion vectors of two control points of the top-leftcorner and top-right corner of a target block to be currently encoded,that is, four parameters, as shown in FIG. 8A. The other model is amodel using the motion vectors of three control points of the top-leftcorner, top-right corner and bottom-left corner of the target block,that is, six parameters, as shown in FIG. 8B.

The four-parameter affine model is expressed as shown in Equation 10.The motion at the sample position (x, y) in the target block may becalculated according to Equation 10. The position of the above leftsample of the target block is assumed to be (0, 0).

$\begin{matrix}\left\{ \begin{matrix}{{mv_{x}} = {{\frac{{mv_{1x}} - {mv_{0x}}}{W}x} - {\frac{{my_{1y}} - {mv_{0y}}}{W}y} + {mv_{0x}}}} \\{{mv_{y}} = {{\frac{{mv_{1y}} - {mv_{0y}}}{W}x} + {\frac{{mv_{1x}} - {my_{0x}}}{W}y} + {mv_{0y}}}}\end{matrix} \right. & {{Equation}10}\end{matrix}$

The six-parameter affine model is expressed by Equation 11. The motionat the sample position (x, y) in the target block may be calculatedaccording to Equation 11.

$\begin{matrix}\left\{ \begin{matrix}{{mv}_{x} = {{\frac{{mv}_{1x} - {mv}_{0x}}{W}x} + {\frac{{mv_{2x}} - {mv_{0x}}}{W}y} + {mv_{0x}}}} \\{{mv_{y}} = {{\frac{{mv_{1y}} - {mv_{0y}}}{W}x} + {\frac{{mv_{2y}} - {mv_{0y}}}{W}y} + {mv_{0y}}}}\end{matrix} \right. & {{Equation}11}\end{matrix}$

wherein, (mv_(0x), mv_(0y)) is the motion vector of the top-left cornercontrol point, (mv_(1x), mv_(1y)) is the motion vector of the top-rightcorner control point, and (mv_(2x), mv_(2y)) is the motion vector of thebottom-left corner control point. Additionally, W is the horizontallength of the target block, and H is the vertical length of the targetblock.

The affine motion prediction may be performed for each sample in thetarget block using a motion vector calculated according to Equation 10or 11. Alternatively, to reduce the computational complexity, theprediction may be performed for each subblock split from the targetblock as illustrated in FIG. 9. For example, the size of the subblockmay be 4×4, 2×2, or 8×8. In an exemplary embodiment below, the affinemotion prediction for the target block is performed on a 4×4 subblockbasis. This example is for convenience of explanation only and thepresent disclosure is not limited thereto.

In the subblock-based affine motion prediction, the motion vector(affine motion vector) of each subblock is calculated by substitutingthe center position of each subblock in (x, y) of Equation 10 or 11. Inparticular, the center position may be an actual center point of thesubblock or the bottom-right sample position of the center point. Forexample, in the case of a 4×4 subblock in which the coordinates of thebottom-left sample are (0, 0), the center position of the subblock maybe (1.5, 1.5) or (2, 2). The predicted block for each subblock isgenerated using the affine motion vector (mv_(x), mv_(y)) of thesubblock. The motion vectors (mv_(x), mv_(y)) may be set to have 1/16sample precision. In particular, the motion vector (mv_(x), mv_(y))calculated according to Equation 1 or 2 may be rounded in 1/16 sampleunits.

The affine motion prediction may be performed not only for the lumacomponent but also for the chroma component. In the case of a 4:2:0video format, when affine motion prediction for the luma component isperformed on a 4×4 subblock basis, affine motion prediction for thechroma component may be performed on a 2×2 subblock basis. The motionvector (mv_(x), mv_(y)) of each subblock of the chroma component may bederived from the motion vector of the corresponding luma component.Alternatively, the size of the subblock for affine motion prediction ofthe chroma component may be the same as that for the luma component.

When affine motion prediction for the luma component is performed on a4×4 subblock basis, affine motion prediction for the chroma component isalso performed on the 4×4 subblock basis. In particular, since the 4×4subblock for the chroma component corresponds to four 4×4 subblocks forthe luma component, the motion vector (mv_(x), mv_(y)) for the subblockof the chroma component may be calculated by calculating the average ofthe motion vectors of the four corresponding subblocks of the lumacomponent.

The video encoding apparatus is configured to perform intra-prediction,inter-prediction (translational motion prediction), affine motionprediction, and the like, and calculates a rate-distortion (RD) cost toselect an optimal prediction method. To perform affine motionprediction, the inter-predictor 124 of the video encoding apparatusdetermines which of the two types of models to use, and determines twoor three control points according to the determined type. Theinter-predictor 124 is configured to calculate the motion vector(mv_(x), mv_(y)) for each of 4×4 subblocks in the target block using themotion vectors of the control points. Then, the inter-predictor 124 isconfigured to perform motion compensation in a reference picture on asubblock-by-subblock basis using the motion vector (mv_(x), mv_(y)) ofeach subblock to generate a predicted block for each subblock in thetarget block.

The entropy encoder 155 of the video encoding apparatus is configured toencode affine related syntax elements including a flag indicatingwhether affine motion prediction is applied to the target block, typeinformation indicating a type of the affine model, and motioninformation indicating a motion vector of each control point, andtransmit the same to the video decoding apparatus. The type informationand the motion information about the control points may be signaled whenaffine motion prediction is performed, and as many motion vectors of thecontrol points as a number determined according to the type informationmay be signaled.

The video decoding apparatus is configured to determine the type of theaffine model and the control point motion vectors using the signaledsyntax elements, and calculate the motion vector (mv_(x), mv_(y)) foreach 4×4 subblock in the target block using Equation 10 or 11. When themotion vector resolution information about the affine motion vector ofthe target block is signaled, the motion vector (mv_(x), mv_(y)) iscorrected to a precision identified by the motion vector resolutioninformation through operations such as rounding-off.

The video decoding apparatus is configured to generate a predicted blockfor each subblock by performing motion compensation within a referencepicture using the motion vector (mv_(x), mv_(y)) for each subblock. Toreduce the amount in bits required to encode the motion vectors ofcontrol points, a method as used in the above-described normalinter-prediction (translation motion prediction) may be applied.

As an example, in the merge mode, the inter-predictor 124 of the videoencoding apparatus is configured to derive a motion vector of eachcontrol point from neighboring blocks of the target block. For example,the inter-predictor 124 is configured to generate a merge candidate listby deriving a predefined number of merge candidates from neighboringsamples L, BL, A, AR, and AL of the target block illustrated in FIG. 4.Each of the merge candidates included in the list corresponds to a pairof motion vectors of two or three control points.

First, the inter-predictor 124 is configured to derive a merge candidatefrom the control point motion vectors of the neighboring blockspredicted in the affine mode among the neighboring blocks. In someembodiments, the number of merge candidates derived from neighboringblocks predicted in the affine mode may be limited. For example, theinter-predictor 124 may be configured to derive two merge candidatesfrom the neighboring blocks predicted in the affine mode: one of L andBL and one of A, AR, and AL. The priority may be assigned in order of Land BL and in order of A, AR and AL. When the total number of mergecandidates is greater than or equal to 3, the inter-predictor 124 may beconfigured to derive a necessary number of merge candidates from thetranslational motion vectors of the neighboring blocks.

FIG. 10 is an exemplary diagram illustrating a method of deriving mergecandidates for affine motion prediction from translation motion vectorsof neighboring blocks. The inter-predictor 124 is configured to derivecontrol point motion vectors CPMV1, CPMV2, CPMV3 from a neighboringblock group {B2, B3, A2}, a neighboring block group {B1, B0}, and aneighboring block group {A1, A0}, respectively. As an example, thepriorities in each neighboring block group may be assigned in order ofB2, B3, A2, order of B1 and B0, and order of A1 and A0. In addition,another control point motion vector CPMV4 is derived from a collocatedblock Tin the reference picture. The inter-predictor 124 is configuredto generate as many merge candidates as necessary through combinationsof two or three control point motion vectors among the four controlpoint motion vectors. The priorities of the combinations are assigned asshown below. The elements in each group are listed in order of thetop-left corner, top-right corner, and bottom-left corner control pointmotion vectors.

-   -   {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3,        CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

The inter-predictor 124 is configured to select a merge candidate in themerge candidate list and perform affine motion prediction on the targetblock. When the selected candidate consists of two control point motionvectors, affine motion prediction is performed using a four-parametermodel. On the other hand, when the selected candidate consists of threecontrol point motion vectors, affine motion prediction is performedusing a six-parameter model. The entropy encoder 155 of the videoencoding apparatus encodes index information indicating the selectedmerge candidate among the merge candidates in the merge candidate listand signals the same to the video decoding apparatus.

The entropy decoder 510 of the video decoding apparatus decodes theindex information signaled from the video encoding apparatus. Theinter-predictor 544 of the video decoding apparatus constructs a mergecandidate list in the same manner as the video encoding apparatus, andis configured to perform affine motion prediction using control pointmotion vectors corresponding to the merge candidate indicated by theindex information.

As another example, in the AMVP mode, the inter-predictor 124 of thevideo encoding apparatus is configured to determine the type of theaffine model and control point motion vectors for the target block.Then, the inter-predictor 124 is configured to determine motion vectordifferences, which are differences between the actual control pointmotion vectors of the target block and the predicted motion vectors ofthe respective control points, and transmit the motion vectordifferences which respectively correspond to the control points.Accordingly, the inter-predictor 124 of the video encoding apparatusconfigures a list of a predefined number of affine AMVPs. When thetarget block is of a 4-parameter type, the candidates included in thelist are each composed of a pair of two control point motion vectors. Onthe other hand, when the target block is of a 6-parameter type, thecandidates included in the list are each composed of a pair of threecontrol point motion vectors. The affine AMVP list may be derived usingcontrol point motion vectors or translational motion vectors ofneighboring blocks in a manner similar to the method of constructing amerge candidate list described above.

However, to derive candidates to be included in the affine AMVP list,there may be a restriction of considering only neighboring blocks thatrefer to the same reference picture as the target block among theneighboring blocks of FIG. 4.

In addition, in the AMVP mode, the affine model type of the target blockshould be considered. When the affine model type of the target block isa 4-parameter type, the video encoding apparatus derives two controlpoint motion vectors (top-left corner and top-right corner control pointmotion vectors of the target block) using the affine model of aneighboring block. When the affine model type of the target block is a6-parameter type, the apparatus derives three control point motionvectors (top-left corner, top-right corner and bottom-left control pointmotion vectors of the target block) using the affine model of aneighboring block.

When the neighboring block is of the 4-parameter type, two or threecontrol point motion vectors are predicted according to the affine modeltype of the target block using two control point motion vectors of theneighboring block. For example, an affine model of a neighboring blockexpressed by Equation 10 may be used. In Equation 10, (mv_(0x), mv_(0y))and (mv_(1x), mv_(1y)) are replaced by the top-left corner and top-rightcorner control point motion vectors of the neighboring block,respectively. Additionally, W is replaced by the horizontal length ofthe neighboring block. The predicted motion vector for each controlpoint of the target block may be derived by inputting, into (x, y), thedifference between the position of the corresponding control point ofthe target block and the position of the top-left corner of theneighboring block.

When the neighboring block is of the 6-parameter type, two or threecontrol point motion vectors are predicted according to the affine modeltype of the target block using three control point motion vectors of theneighboring block. For example, an affine model of a neighboring blockexpressed by Equation 11 may be used. In Equation 11, (mv_(0x),mv_(0y)), (mv_(1x), mv_(1y)), and (mv_(2x), mv_(2y)) are replaced bycontrol point motion vectors of the top-left corner, top-right cornerand bottom-left corner of the neighboring block, respectively. Thepredicted motion vector for each control point of the target block maybe derived by inputting, into (x, y), the difference between theposition of the corresponding control point of the target block and theposition of the top-left corner of the neighboring block.

The inter-predictor 124 of the video encoding apparatus is configured toselect one candidate in the affine AMVP list, and generate motion vectordifferences between the motion vector of each actual control point andthe predicted motion vector of the corresponding control point of theselected candidate. The entropy encoder 155 of the video encodingapparatus is configured to encode type information indicating the affinemodel type of the target block, index information indicating a selectedcandidate among the candidates in the affine AMVP list, and a motionvector difference corresponding to each control point and transmits thesame to the video decoding apparatus.

The inter-predictor 544 of the video decoding apparatus is configured todetermine the affine model type using the information signaled from thevideo encoding apparatus, and generate a motion vector difference ofeach control point. Then, the inter-predictor is configured to generatean affine AMVP list in the same manner as the video encoding apparatus,and select a candidate indicated by the index information signaled inthe affine AMVP list. The inter-predictor 544 of the video decodingapparatus is configured to calculate a motion vector of each controlpoint by adding the predicted motion vector of each control point of theselected candidate and a corresponding motion vector difference.

III. Sample-By-Sample Adjustment of Affine Motion Prediction Samples

Subblock-by-subblock affine motion prediction for a target block hasbeen described above. Another aspect of the present disclosure relatesto adjusting, on a sample-by-sample basis, sample values of predictedsamples generated by subblock-by-subblock affine motion prediction. Themotion according to the position of each sample is additionallycompensated in each subblock that forms the basis of affine motionprediction.

When the sample values in any one subblock generated as a result ofsubblock-by-subblock affine motion prediction for a target block areI(x, y), the video encoding apparatus calculates the horizontal andvertical gradients g_(x)(i, j) and g_(y)(i, j) at each sample position.Equation 12 may be used to calculate the gradients.

g _(x)(i,j)=I(i+1,j)−I(i−1,j)

g _(y)(i,j)=I(i,j+1)−I(i,j−1)  Equation 12

The sample offset ΔI(i, j) for adjusting the prediction sample iscalculated by the following equation.

ΔI(i,j)=g _(x)(i,j)*Δmv _(x)(i,j)+g _(y)(i,j)*Δmv _(y)(i,j)  Equation 13

wherein, Δmv(i,j) denotes a motion offset, that is, a difference betweenthe affine motion vector in the sample (i, j) and the affine motionvector at the center position of the subblock, and may be calculated byapplying Equation 10 or 11 depending on the affine model type of thetarget block. In other words, Δmv(i, j) may be calculated from Equation10 or 11 by subtracting the motion vector given when the subblock centerposition is input to (x, y) from the motion vector given when (i, j) isinput to (x, y). Additionally, Δmv(i, j) may be calculated fromEquations 11 and 12 except for the last term “+mv_(0x)” and “+mv_(0y)”by substituting (x, y) in Equations 11 and 12 with the horizontal andvertical offsets from the subblock center position to the sampleposition (i,j). The center position may be the actual center point ofthe subblock, or may be the bottom-right sample position of the centerpoint.

The motion vector of each control point of the target block used forcalculation of Δmv(i, j) and the difference between the sample position(i, j) and the center position of the subblock are the same for allsubblocks. Accordingly, the values of Δmv(i, j) may be calculated foronly one subblock, for example, a first subblock, and may be reused forthe other subblocks.

The technique of the present disclosure is based on the assumption thatthe object moves at a constant speed and the change in sample values issmooth. Therefore, sample variations in the horizontal direction and theamount of sample change in the vertical direction are obtained bymultiplying the x component (Δmv_(x)) and y component (Δmv_(y)) ofΔmv(i, j) by the horizontal and vertical sample gradient values,respectively. The sample offset ΔI(i,j) is calculated by adding the twosample variations. The final value of the predicted sample is calculatedas follows.

I′(i,j)=I(i,j)+ΔI(i,j)  Equation 14

When sample-by-sample adjustment for affine motion prediction samples isapplied, the inter-predictor 124 of the video encoding apparatus and theinter-predictor 544 of the video decoding apparatus perform theabove-described process to modifying the sample values of the predictedsamples generated through the affine motion prediction. Gradient valuesare derived from the predicted samples generated through the affinemotion prediction, and Δmv(i, j) is derived from the control pointmotion vectors of the target block. Therefore, it is not required thatthe video encoding apparatus signals additional information for thepresent technique to the video decoding apparatus.

IV. Motion Vector Refinement

As described above, in the case of inter-prediction using bi-directionalprediction, a reference picture Ref₀ selected from a reference picturelist L0, a motion vector MV₀ referring to the reference picture Ref₀, areference pictures Ref₁ selected from a reference picture list L1, and amotion vector MV₁ referring to the reference picture Ref₁ aredetermined. The motion vector refinement described in this embodiment isa coding tool to improve the accuracy of inter-prediction by finelyadjusting the two motion vectors MV₀ and MV₁ for bi-directionalprediction. The motion vector refinement may be applied to a merge modein which a motion vector of a neighboring block is used as a motionvector of a target block. Alternatively, it may be applied to an AMVPmode (normal inter-prediction) or an affine mode.

FIG. 11 is an exemplary diagram illustrating the concept of a motionvector refinement coding tool according to an embodiment of the presentdisclosure. In bi-directional prediction, the inter-predictor 124 of thevideo encoding apparatus is configured to search for a refined motionvector around the initial motion vectors for the reference picture Ref₀and the reference picture Ref₁. In particular, the initial motionvectors represent the two motion vectors MV₀ and MV₁ for bi-directionalprediction. Distortion between two candidate blocks in the referencepictures Ref₀ and Ref₁ is calculated using block matching. As shown inFIG. 11, in the reference pictures Ref₀ and Ref₁, a sum of absolutedifference (SAD) between two candidate blocks at the positionsdetermined by the respective initial motion vectors and between the twocandidate blocks around the positions is calculated.

In other words, the sum of the absolute values of the difference betweenthe sample values corresponding to each other in both candidate blocksis calculated. The inter-predictor 124 is configured to select a motionvector candidate having the minimum SAD as a refined motion vector anduses the same to generate a bi-directional prediction signal accordingto inter-prediction. While the SAD is used to represent the differencebetween the two candidate blocks, embodiments are not limited thereto.Any metric capable of expressing the difference between the twocandidate blocks, such as the sum of squared error (SSE) may be used.

Motion vector refinement may be applied when one reference pictureprecedes the current picture in display order, the other referencepicture follows the current picture, and the distances from the tworeference pictures to the current picture are the same, that is, thereference pictures have the same difference in picture order count (POC)from the current picture. As shown in FIG. 11, for two initial motionvectors MV₀ and MV₁ in the two reference pictures Ref₀ and Ref₁, refinedbi-directional motion vector candidates MV₀′ and MV₁′ in the respectivereference picture lists can be expressed as in Equation 15.

MV₀′=MV₀+MV_offset

MV₁′=MV₁−MV_offset  Equation 15

wherein, MV_offset is an offset for the initial motion vectors accordingto motion vector refinement, and represents a difference between amotion vector candidate and a corresponding initial motion vector. Thisoffset may be a sum of an integer offset in units of integer samples anda subpixel offset in units of sub-pixel or sub-pel samples. As shown inEquation 15, candidates for the two motion vectors conform to amirroring rule for the MV difference. The process of searching for anoptimal offset for a refined motion vector includes two operations: aninteger sample search for searching for an integer offset, and afractional sample refinement for searching for a sub-pixel offset.

First, in the integer sample search operation, the inter-predictor 124is configured to search for all 25 neighboring points, including thecenter of refinement as shown in FIG. 12. In particular, the center ofthe refinement corresponds to the position indicated by the initialmotion vector. When the SAD between a pair of two candidate blocks atthe two positions determined according to the initial motion vectors isless than a specific threshold, the operation after the integer samplesearch may be skipped. By calculating the SAD between the two candidateblocks corresponding to the neighboring points of the positionsindicated by the initial motion vectors, the inter-predictor 124 isconfigured to search for a point having the minimum SAD, that is,minSAD. According to the result of the integer sample search, theinteger offset for the motion vector candidate having minSAD isexpressed as (intOffX, intOffY).

After the integer sample search operation is completed, when intOffX ofthe integer offsets is not −2 or 2 and intOffY is not −2 or 2{(intOffX!=(2∥−2)) && (intOffY!=(2∥−2))}, the inter-predictor 124 setssubPelFlag, which is an indication flag for applying the fractionalsample refinement operation, to a value of ‘true’. When subPelFlag isset to ‘true’, the inter-predictor 124 performs the fractional samplerefinement operation. The inter-predictor 124 is configured to generatean optimal sub-pixel offset by solving a parametric error surfaceequation in a two-dimensional space as shown in Equation 16.

E(x,y)=A(x−x _(min))² +B(y−y _(min))² +C  Equation 16

wherein, E(x,y) is a cost at position (x,y) and may be defined as an SADbetween two candidate blocks corresponding to position (x,y).Additionally, (x_(min), y_(min)) is a sample position with a minimumcost C and represents an optimal sub-pixel offset.

On the assumption that the sample position corresponding to the integeroffset according to the integer sample search is (0,0), an optimalsub-pixel offset may be calculated with the cost of blocks correspondingto all four positions (top, bottom, left and right positions), (−1,0),(1,0), (0,1), and (0,1), as shown in Equation 17.

x _(min)=(E(−1,0)−E(1,0))/(2(E(−1,0)+E(1,0)−2E(0,0)))

y _(min)=(E(0,−1)−E(0,1))/(2(E(0,−1)+E(0,1)−2E(0,0)))  Equation 17

The constraint on the cost E(x,y) may be expressed by Equation 18.

E(x,y)≥0 && E(0,0) is minimum  Equation 18

Based on Equation 18, ‘−0.5<x_(min), y_(min)<0.5’ is satisfied.

The inter-predictor 124 may be configured to generate an optimal offsetin a sub-pixel unit by adding the sub-pixel offset and the integeroffset, and then add the optimal offset to the initial motion vectorsusing Equation 15 to obtain refined bi-directional motion vectors. Inaddition, the inter-predictor 124 may be configured to generateinter-predicted samples for the target block using the refinedbi-directional motion vectors and the bi-directional reference pictures.

As described above, in the motion vector refinement process, motionvectors are refined using motion information (two motion vectors and tworeference pictures) used for bi-directional prediction. Accordingly, theinter-predictor 544 of the video decoding apparatus may also beconfigured to perform motion vector refinement in the same manner as thevideo encoding apparatus using the motion information (motion vectorsand reference pictures) used for bi-directional prediction and receivedfrom the video encoding apparatus. Therefore, signaling of additionalinformation for the motion vector refinement process from the videoencoding apparatus to the video decoding apparatus is not required.

When the motion vector refinement process is applied, various complexityissues may be raised in particular at the decoder side because SADvalues for candidate blocks must be calculated for every target block.When the fractional sample refinement operation is performed for motionvectors based on a sample corresponding to an integer offset, a complexequation as shown in Equation 16 needs to be solved. This process may bedifficult to optimize with hardware. When the merge mode is suitable forthe inter-prediction, the need for the fractional sample refinementoperation may be relatively low in motion vector refinement.Accordingly, the motion vector refinement process for each of blocksencoded in the merge mode increases the complexity of the videoencoding/decoding apparatus and makes it difficult to expectcorresponding compression efficiency.

In view of the above, the present disclosure provides a method forreducing the complexity of the video encoding/decoding apparatus bydetermining the need for adjustment of motion vectors on a sub-pixelbasis using the integer offset information acquired in the integersample search operation and skipping the fractional sample refinementoperation according to the result of the determination. In the motionvector refinement process described above, when the SAD of the initialmotion vector pairs is greater than or equal to a specific threshold,the integer sample search and fractional sample refinement operationsare performed. The threshold is determined according to the size of thetarget block. For example, it may be set as the value ofsbHeight×sbWidth, that is, the area of the target block.

In another example according to the present embodiment, the thresholdmay be determined based on a quantization parameter as well as the areaof the target block. In another example, whether to perform the integersample search and fractional sample refinement operations may bedetermined based on an index value of a neighboring block searched inthe merge mode. In the integer sample search operation of the motionvector refinement process, the search operation is performed on a totalof 25 integer coordinates horizontally and vertically from −2 to +2,including the center of the refinement corresponding to the initialmotion vector, as shown in FIG. 12. Accordingly, the integer offset forthe position of the block with minSAD is expressed as (intOffX,intOffY).

To skip the complex equation solving process according to thesub-pixel-based motion vector refinement operation as shown in Equation16 and Equation 17, the fractional sample refinement operation may beterminated early after performing a search up to an integer sample of aspecific condition in the integer sample search operation. When bothintOffX and intOffY are 0, the inter-predictor 124 of the video encodingapparatus may be configured to skip the fractional sample refinementoperation of searching for a sub-pixel offset, and set the sub-pixeloffset as shown in Equation 17 to zero. Alternatively, when (intOffX,intOffY) is not (0, 0), the inter-predictor 124 may set subPelFlag tothe value of ‘true’ in order to apply the fractional sample refinementoperation. Thereby, when (intOffX, intOffY) is (0, 0), the fractionalsample refinement operation may be terminated early.

As another aspect of an embodiment of the present disclosure, when theSAD acquired in the integer sample search operation is less than aspecific threshold, the inter-predictor 124 is configured to skip thefractional sample refinement operation of searching for sub-pixeloffsets for vectors. For example, in addition to minSAD, let the secondand third smallest SADs be denoted as SAD_1 and SAD_2, respectively.When the condition expressed as Equation 19 are satisfied, thefractional sample refinement operation may be skipped.

Abs(minSAD−SAD_1)<threshold && Abs(minSAD−SAD_2)<threshold  Equation 19

wherein, Abs(⋅) is a function that changes an input to an absolutevalue.

As another aspect of the embodiment according to the present disclosure,the inter-predictor 124 is configured to determine whether to skip thefractional sample refinement operation of searching for a sub-pixeloffset for a motion vector, based on the magnitude of the motion vectorof a neighboring block searched for in the merge mode. For example, whentwo or more motion vectors of two or more neighboring blocks searchedfor in the merge mode are the same or similar (e.g., the differencebetween the motion vectors is less than a certain threshold), thefractional sample refinement operation may be skipped.

In the merge mode, based on the magnitude of the motion vector searchedfor at one or more specific indexes, the inter-predictor 124 isconfigured to determine whether to skip the fractional sample refinementoperation of searching for a sub-pixel offset for the motion vector. Forexample, the fractional sample refinement operation may be skipped inthe merge mode, when the values of motion vectors searched for in thetop left, bottom left, and top right blocks are equal or have littledifference from each other. As another example, when the magnitudes ofthe motion vectors searched for in the top left, bottom left, and topright blocks are equal to 0 or less than a specific threshold, thefractional sample refinement operation may be skipped.

In the merge mode, the inter-predictor 124 is configured to determinewhether to skip the fractional sample refinement operation of searchingfor a sub-pixel offset for a motion vector, based on a process thatmotion vectors searched for at one or more specific indexes have beenderived. For example, when the values of motion vectors of the top left,bottom left, and top right blocks have been derived without the motionvector refinement process, the motion vector refinement process is notadditionally performed in the current block. As another example, whenthe fractional sample refinement operation of the motion vectorrefinement process has not been used in the process of deriving themotion vectors of the top left, bottom left, and top right blocks, thefractional sample refinement operation may be skipped in the motionvector refinement process for the current block.

When one or more neighboring blocks (e.g., top left, bottom left, andtop right blocks) of the current block use the merge mode and is encodedin the skip mode (i.e., cu_skip_flag==TRUE for the blocks), the imageencoding/decoding apparatus is configured to determine that the need forprecision of the motion vector of the current block is low, and thusskips the motion vector refinement process. In particular, in skippingthe motion vector refining process, only the fractional samplerefinement operation may be skipped, or the integer sample searchoperation including the fractional sample refinement operation may beskipped.

As described above, in the motion vector refinement process according tothe present embodiment, the search complexity of the inter-predictor 124may be reduced by terminating the fractional sample refinement operationearly.

The inter-predictor 124 may be configured to perform early terminationof the motion vector refinement process, based on the motion information(two motion vectors and two reference pictures) used for prediction,motion vectors of neighboring blocks searched for in the merge mode, andthe like. Accordingly, the inter-predictor 544 of the video decodingapparatus may also be configured to perform early termination of themotion vector refinement process in the same manner as in the videoencoding apparatus, based on the motion information (motion vectors andreference pictures) used for bi-directional prediction received from thevideo encoding apparatus, motion vectors of neighboring blocks searchedfor in the merge mode, and the like. Accordingly, signaling ofadditional information for early termination of the motion vectorrefinement process from the video encoding apparatus to the videodecoding apparatus is not required.

Hereinafter, as another aspect of the embodiment according to thepresent disclosure, a method for performing an additional fractionalsample refinement operation at the boundary of a search range for themotion vector refinement process will be described.

As described above, when intOffX is −2 or 2 or intOffY is −2 or 2{(intOffX==(2−2))∥(intOffY=(2∥−2))}, the fractional sample refinementoperation of the motion vector refinement process is skipped. Since theresult of the motion vector refinement is limited to −1.5 to 1.5according to the condition shown in Equation 18, the compressionperformance of the video encoding/decoding apparatus may be limited.Accordingly, the motion vector refinement according to the presentdisclosure may allow the fractional sample refinement operation for amotion vector in units of sub-pixels from −2 to −1.5 and from 1.5 to 2while limiting the range of integer sample search to −2 to 2, therebyimproving compression performance.

Even when intOffX is −2 or 2 or intOffY is −2 or 2, the inter-predictor124 of the video encoding apparatus may be configured to set subPelFlagto the value of TRUE in order to apply the fractional sample refinementoperation. When subPelFlag is set to TRUE, the inter-predictor 124 maybe configured to execute a fractional sample refinement operation ofsearching for a sub-pixel offset in units of at least ¼ pel to executemotion vector refinement for coordinates less than or equal −1.75 orgreater than or equal to 1.75. As another example, when subPelFlag isset to the value of TRUE, the inter-predictor 124 may be configured toexecute a fractional sample refinement operation of searching for asub-pixel offset in units of at least ⅛ pel to execute motion vectorrefinement for the coordinate of −1.875 or 1.875.

Alternatively, by deleting the subPelFlag, the inter-predictor 124 maybe configured to execute the fractional sample refinement operation ofsearching for a sub-pixel offset even when intOffX is −2 or 2 or intOffYis −2 or 2. As described above, when intOffX is −2 or 2 or intOffY is −2or 2, the inter-predictor 124 may be configured to execute thefractional sample refinement operation of searching for a sub-pixeloffset in units of at least ¼ pel to refine the motion vector for thecoordinate of −1.75 or 1.75. In addition, when intOffX is −2 or 2 orintOffY is −2 or 2, the motion vector refinement for the coordinate of−1.25 or 1.25 is not performed.

To apply Equation 17, 4 positions of the top, bottom, left, and rightare required for (intOffX, intOffY). However, when intOffX is −2 or 2 orintOffY is −2 or 2, there may not exist one or more of the requiredpositions. Accordingly, 4 positions of the top, bottom, left, and rightfor (intOffX, intOffY) may be set by applying Max(−2, intOffX−1),intOffX, and Min(intOffX+1, 2) in the horizontal direction and applyingMax(−2, intOffY−1), intOffY, and Min(intOffY+1, 2) in the verticaldirection. In other words, when (intOffX, intOffY) is positioned at theoutermost side of 25 points, The outermost point (intOffX, intOffY) maybe used as the top, bottom, left, or right position.

The motion vector refinement coding tool described above requires theinteger sample search to be performed through SAD calculation for 25sample positions determined by the initial motion vector, and thefractional sample refinement to be performed according to the parametricerror-surface equation, and therefore it may increase complexity.Hereinafter, a simplified motion vector refinement process will bedescribed. The simplified motion vector refinement process may reducethe complexity of the video encoding/decoding apparatus when thefractional sample refinement operation is additionally performed,including even the case where intOffX is −2 or 2 or intOffY is −2 or 2.

In an embodiment of the present disclosure, as shown in FIG. 12, theinter-predictor 124 is configured to calculate the SADs for positionsA1, A2, A3, and A4 in the integer sample search operation to find aposition having minSAD. When the position having minSAD is A4, the SADsfor positions A41, A42, A43, and A44 around A4 are calculated. Based onthe calculated SADs, the inter-predictor 124 is configured to executethe fractional sample refinement operation using the parametricerror-surface equation as shown in Equation 16 and Equation 17, therebygenerating (x_(min), y_(min)) in units of ⅛ pel. Since theabove-described process does not satisfy the constraint that the cost ofthe center point shall be the minimum among the constraints shown inEquation 18, all positions within the search range may be searched forin units of ⅛ pel. This search method reduces the number of searches inthe integer sample search operation from 25 to 8, and therefore thesearch complexity may be significantly reduced.

In another embodiment of the present disclosure, as shown in FIG. 13,the inter-predictor 124 is configured to calculate the SADs forpositions A1, A2, A3, A4, and A5 in the integer sample search operation,and execute the fractional sample refinement operation based on thecalculated SADs, thereby generating (x_(min), y_(min)) in units of ¼pel. The approximate position of (x_(min), y_(min)) is checked among thefirst to fourth quadrants. Then, when it is positioned in, for example,the second quadrant, the inter-predictor 124 is configured toadditionally calculate SADs for positions B1, B2, B3, B4, and B5. Basedon the calculated SADs, the inter-predictor 124 is configured toadditionally perform a fractional sample refinement operation togenerate (x_(min), y_(min)) in units of ⅛ pel. The above process doesnot satisfy the constraint that the cost of the center point shall bethe minimum among the constraints shown in Equation 18, and accordinglyall positions within the search range may be searched for in units of ⅛pel. With this search method, the search for 25 sample positionsrequired in the integer sample search operation is replaced with thesearch for 10 sample positions and an additional fractional samplerefinement. Thereby, complexity may be reduced.

In another embodiment of the present disclosure, as shown in FIG. 13,the inter-predictor 124 is configured to calculate SADs for positionsA1, A2, A3, A4, and A5 in the integer sample search operation, andexecute the fractional sample refinement operation based on thecalculated SADs to generate (x_(min), y_(min)) in units of ¼ pel. Sincethe above process does not satisfy the constraint that the cost of thecenter point shall be the minimum among the constraints shown inEquation 18, all positions within the search range may be searched forin units of ⅛ pel. With this search method, the number of searches inthe integer sample search operation may be reduced from 25 to 5.Thereby, the search complexity may be significantly reduced.

In another embodiment of the present disclosure, as shown in FIG. 14,the inter-predictor 124 is configured to calculate SADs for positions A1to A9 in the integer sample search operation to find a position havingminSAD. For example, when A5 is a position having minSAD, theinter-predictor 124 is configured to execute the fractional samplerefinement operation based on the SADs pre-calculated for positions A2,A4, A6, and A8 to generate (x_(min), y_(min)) in units of ⅛ pel. Sincethe above-described process does not satisfy the constraint that thecost of the center point shall be the minimum among the constraintsshown in Equation 18, all positions within the search range may besearched for in units of ⅛ pel.

When A7 is a position having minSAD according to the result of executingthe integer sample search, SADs for positions B1 and B2 are additionallycalculated. Based on the SADs for positions A4, A7, A8, B1, and B2, theinter-predictor 124 generates (x_(min), y_(min)) in units of ⅛ pel byexecuting the fractional sample refinement operation. Since theabove-described process does not satisfy the constraint that the cost ofthe center point shall be the minimum among the constraints shown inEquation 18, all positions within the search range may be searched forin units of ⅛ pel. With this search method, the number of searches inthe integer sample search operation may be reduced from 25 to a minimumof 8 and a maximum of 11. Thereby, search complexity may be reduced.

As described above, in the motion vector refinement process according tothe present embodiment, the compression performance may be increased byextending the search range for the sub-pixel offset, and searchcomplexity may be reduced by simplifying the integer sample search.

The video encoding apparatus may extend the search range of the motionvector refinement process and simplify the search operation, using themotion information (two motion vectors and two reference pictures) usedfor prediction. Therefore, the inter-prediction unit 544 of the videodecoding apparatus may also extend the search range of the motion vectorrefinement process and simplify the search operation in the same manneras in the video encoding apparatus, using the motion information (motionvectors and reference pictures) for bi-directional prediction receivedfrom the video encoding apparatus. Therefore, signaling of additionalinformation from the video encoding apparatus to the video decodingapparatus to simplify the search operation of the motion vectorrefinement process is not required.

V. High-Level Syntax Signaling

Another aspect of the present disclosure relates to a technique forsignaling of high-level syntaxes for controlling on/off of various toolsdescribed above. The above-described optical flow, sample-by-sampleadjustment for affine motion prediction samples, and motion vectorrefinement are coding tools used to improve the video encodingefficiency. However, for specific content such as, for example, screencontent, the aforementioned various coding tools may not contribute toimproving compression performance. Accordingly, a coding unit basedsignaling of whether to apply each coding tool is applied or a codingunit based decision of whether to apply the tool may rather degradecoding efficiency or increase computational complexity. The presentdisclosure provides a signaling technique for efficiently controllingthe above-described coding tools.

To encode a sequence of pictures, the video encoding apparatus isconfigured to encode a first high-level syntax element of the sequencelevel indicating whether each of the above-described a plurality ofcoding tools is disabled. The first high-level syntax element may beencoded in the header of the sequence, that is, the SPS. In particular,the plurality of coding tools includes at least one of optical flow,sample-by-sample adjustment, and motion vector refinement.

When the first high-level syntax element indicates that each of theplurality of coding tools is not enabled, the corresponding coding toolis applied to the encoding of the sequence. In other words, each of thecoding tools described above is not applied to any pictures in thesequence. When the first high-level syntax element indicates that eachof a plurality of coding tools is enabled, each of the plurality ofcoding tools may be used to encode the corresponding sequence. Forexample, optical flow is applicable to all pictures in the sequence.Additionally, motion vector refinement is applicable to all pictures inthe sequence.

The video encoding apparatus is configured to encode a picture levelflag indicating whether a corresponding coding tool is allowed in apicture header (i.e., PPS) of each of the pictures belonging to thesequence depending on the value of the first high-level syntax element.For example, when the first high-level syntax element indicates that thecorresponding coding tool is allowed at the sequence level, the picturelevel flag for indicating whether the corresponding coding tool isallowed at the picture level is encoded. When the first high-levelsyntax element indicates that the corresponding coding tool is notallowed at the sequence level, the picture level flag is not separatelycoded. In particular, the video decoding apparatus is configured toautomatically set the picture level flag to a value indicating that thecorresponding coding tool is not allowed.

The video decoding apparatus may be configured to set, at the picturelevel, whether to allow a corresponding coding tool from the picturelevel flag signaled from the video encoding apparatus or a picture levelflag whose value is automatically set. When the picture level flags forthe coding tools according to the present embodiment are not containedin the bitstream, the video decoding apparatus sets each picture levelflag to a value indicating that the corresponding coding tools are notallowed at the picture level.

According to the above-described syntax structure, the video encodingapparatus may be configured to adaptively determine whether to allowcoding tools at the sequence level and the picture level. In addition,the video decoding apparatus may be configured to adaptively determinewhether to allow coding tools at the sequence level and the picturelevel by decoding the syntaxes signaled from the video encodingapparatus.

In some other embodiments of the present disclosure, the picture levelflag of the above-described embodiment may be replaced with a slicelevel flag. In other words, when the first high-level syntax elementindicates that the corresponding coding tool is allowed at the sequencelevel, the video encoding apparatus encodes, in the slice header, aslice level flag for indicating whether the corresponding coding tool isallowed at the slice level. When the first high-level syntax elementindicates that the corresponding coding tool is not allowed, the slicelevel flag is not coded. The video encoding apparatus may be configuredto adaptively determine whether to allow coding tools at the sequencelevel and the slice level. The video decoding apparatus may beconfigured to determine whether to allow each tool at the sequence leveland the slice level according to the first high-level syntax element orthe slice level flag.

In another embodiment of the present disclosure, to encode a sequence ofpictures, the video encoding apparatus is configured to encode the firsthigh-level syntax element of the sequence level indicating whether allthe coding tools as described above are enabled. The first high-levelsyntax element may be encoded in the header of the sequence, that is,the SPS. When the first high-level syntax element of the sequence levelindicates that any of the plurality of coding tools is not enabled, thecorresponding coding tools are not applied to the encoding of thesequence. In other words, none of the coding tools as described above isapplied to any pictures in the sequence. When the first high-levelsyntax element of the sequence level indicates that there is norestriction on use of all the plurality of coding tools, the videoencoding apparatus is configured to encode a second high-level syntaxelement of the slice level indicating whether all the plurality ofcoding tools are enabled. The second high-level syntax element may beencoded in the header of the slice.

When the second high-level syntax element of the slice level indicatesthat any of the plurality of coding tools is not enabled, thecorresponding coding tools are not applied to the encoding of thecorresponding slice. In other words, none of the coding tools asdescribed above is applied to the slice. When the second high-levelsyntax element of the slice level is 0, there is no restriction on useof all the plurality of coding tools. In addition, when the firsthigh-level syntax element of the sequence level indicates that all theplurality of coding tools is disabled, the second high-level syntaxelement of the slice level is set not to enable any of the plurality ofcoding tools, and the corresponding coding tools are not applied to theencoding of the corresponding slice.

In addition, in some other embodiments according to the presentdisclosure, the second high-level syntax element of the slice level asdescribed above may be replaced with the second high-level syntaxelement of the picture level. The second high-level syntax element maybe encoded in the header (i.e., PPS) of the picture. When the secondhigh-level syntax element of the picture level indicates that all theplurality of coding tools is disabled, the corresponding coding toolsare not applied to the encoding of the picture. In other words, any ofthe coding tools as described above is not applied to the picture. Whenthe second high-level syntax element of the picture level is 0, there isno restriction on use of all the plurality of coding tools. In addition,when the first high-level syntax element of the sequence level indicatesthat all the plurality of coding tools is disabled, the secondhigh-level syntax element of the picture level is set not to enable anyof the plurality of coding tools, and the corresponding coding tools arenot applied to the encoding of the picture.

As described above, according to the present embodiment, by providing amethod of simply performing the integer sample search operation and thefractional sample refinement operation in relation to motion vectorrefinement among coding tools for improving the compression performanceof inter-prediction, the coding efficiency of the encoder/decoder may beincreased, and accordingly complexity may be reduced.

Various functions or methods described in the present disclosure may beimplemented with instructions stored in a non-transitory recordingmedium that can be read and executed by one or more processors.Non-transitory recording media include, for example, all types ofrecording devices in which data is stored in a form readable by acomputer system. For example, non-transitory recording media includestorage media such as erasable programmable read only memory (EPROM),flash drives, optical drives, magnetic hard drives, and solid statedrives (SSDs).

Although exemplary embodiments have been described for illustrativepurposes, those skilled in the art will appreciate that and variousmodifications and changes are possible, without departing from the ideaand scope of the embodiments. Exemplary embodiments have been describedfor the sake of brevity and clarity. Accordingly, one of ordinary skillwould understand that the scope of the embodiments is not limited by theembodiments explicitly described above but is inclusive of the claimsand equivalents thereto.

REFERENCE NUMERALS

-   -   124: Inter-predictor    -   155: Entropy encoder    -   510: Entropy decoder    -   544: Inter-predictor

1-23. (canceled)
 24. A video decoding method for decoding a video databy a video decoding apparatus, comprising: decoding, from a bitstream,at least one first high level syntax element indicating whether each ofat least one coding tool is allowed, the at least one coding toolincluding a motion vector refinement; decoding, from the bitstream, asecond high level syntax element corresponding to each of the at leastone coding tool depending on the at least one first high level syntaxelement, and setting, at a picture level, whether each of the at leastone coding tool is allowed; in response to identifying that the motionvector refinement is allowed for a current picture including a targetblock based on the setting at the picture level, generating a refinedmotion vector for the target block, by calculating, for pairs of twocandidate blocks which are within a preset offset range from initialmotion vectors in bi-directional reference pictures of the target block,metrics between the two candidate blocks and searching for a motionvector candidate having a minimum metric; generating inter-predictedsamples for the target block using the bi-directional reference picturesand the refined motion vector; obtaining residual samples for the targetblock; and decoding the target block based on the residual samples andthe inter-predicted samples, wherein the first high level syntax elementis decoded from a header of a sequence.
 25. The method of claim 24,wherein the decoding of the second high level syntax element includes:when the first high level syntax element indicates that the motionvector refinement is allowed, decoding, from a picture header of each ofthe pictures belonging to a sequence, a picture level flag indicatingwhether the motion vector refinement is allowed.
 26. The method of claim25, wherein, when the first high level syntax element indicates that themotion vector refinement is not allowed, the picture level flag is setto a value indicating that the motion vector refinement is not allowedwithout being decoded from the picture header.
 27. The method of claim24, wherein, when the first high level syntax element indicates that themotion vector refinement is not allowed, a coding tool for the motionvector refinement is not allowed for any of the pictures in thesequence.
 28. The method of claim 24, wherein the generating of therefined motion vector includes: an integer sample search operation ofsearching for an integer offset corresponding to a position having theminimum metric by calculating the metrics for pairs of the two candidateblocks which correspond to a pair of initial positions indicated by theinitial motion vectors and pairs of two neighboring positions displacedfrom the initial positions in integer sample units, wherein theneighboring positions in the integer sample units have a squaredistribution around each of the initial positions; a fractional samplerefinement operation of, when the integer offset is not positioned at anoutermost side of the square distribution, searching for an optimalsub-pixel offset by solving a parametric error surface equation usingthe position with the minimum metric and top, bottom, left, and rightneighboring positions of the position with the minimum metric; andgenerating an offset between the initial motion vector and the refinedmotion vector by adding the sub-pixel offset to the integer offset. 29.The method of claim 28, wherein, when the metric between the twocandidate blocks corresponding to the initial motion vectors is lessthan a preset first threshold, both the integer sample search operationand the fractional sample refinement operation are skipped.
 30. Themethod of claim 28, wherein, when the integer offset is zero, thefractional sample refinement operation is skipped.
 31. The method ofclaim 28, wherein when each of differences between the minimum metricand metrics which are acquired at integer coordinates of two or morepositions in the integer sample search operation is less than a presetsecond threshold, the fractional sample refinement operation is skipped.32. The method of claim 28, wherein, when the initial motion vectors forthe target block are generated by a merge mode, whether to execute theinteger sample search operation or the fractional sample refinementoperation is determined depending on magnitudes of motion vectors ofneighboring blocks considered as merge candidates or a coding modeapplied to the neighboring blocks.
 33. A video encoding method forencoding a video data by a video encoding apparatus, comprising:encoding at least one first high level syntax element indicating whethereach of at least one coding tool is allowed, the at least one codingtool including motion vector refinement; encoding a second high levelsyntax element of a picture level related to whether each of the atleast one coding tool is allowed depending on the at least one firsthigh level syntax element; in response to identifying that the motionvector refinement is allowed for a current picture including a targetblock based on setting of the picture level, generating a refined motionvector for the target block, by calculating, for pairs of two candidateblocks which are within a preset offset range from initial motionvectors in bi-directional reference pictures of the target block,metrics between the two candidate blocks and searching for a motionvector candidate having a minimum metric; generating inter-predictionsamples for the target block using the bi-directional reference picturesand the refined motion vector generating residual samples based on thetarget block and the inter-prediction samples; and encoding the residualsamples, wherein the first high level syntax element is decoded from aheader of a sequence.
 34. The method of claim 33, wherein the encodingof the second high level syntax element includes: when the first highlevel syntax element indicates that the motion vector refinement isallowed, encoding, in a picture header of each of the pictures belongingto a sequence, a picture level flag indicating whether the motion vectorrefinement is allowed.
 35. The method of claim 33, wherein thegenerating of the refined motion vector includes: an integer samplesearch operation of searching for an integer offset corresponding to aposition having the minimum metric by calculating the metrics for pairsof the two candidate blocks which correspond to a pair of initialpositions indicated by the initial motion vectors and pairs of twoneighboring positions displaced from the initial positions in integersample units, wherein the neighboring positions in the integer sampleunits have a square distribution around each of the initial positions; afractional sample refinement operation of, when the integer offset isnot positioned at an outermost side of the square distribution,searching for an optimal sub-pixel offset by solving a parametric errorsurface equation using the position with the minimum metric and top,bottom, left, and right neighboring positions of the position with theminimum metric; and generating an offset between the initial motionvector and the refined motion vector by adding the sub-pixel offset tothe integer offset.
 36. The method of claim 35, wherein, when the metricbetween the two candidate blocks corresponding to the initial motionvectors is less than a preset first threshold, both the integer samplesearch operation and the fractional sample refinement operation areskipped.
 37. A non-transitory recording medium storing a bitstreamgenerated by a video encoding method, the method comprising: encoding atleast one first high level syntax element indicating whether each of atleast one coding tool is allowed, the at least one coding tool includingmotion vector refinement; encoding a second high level syntax element ofa picture level related to whether each of the at least one coding toolis allowed depending on the at least one first high level syntaxelement; in response to identifying that the motion vector refinement isallowed for a current picture including a target block based on settingof the picture level, generating a refined motion vector for the targetblock, by calculating, for pairs of two candidate blocks which arewithin a preset offset range from initial motion vectors inbi-directional reference pictures of the target block, metrics betweenthe two candidate blocks and searching for a motion vector candidatehaving a minimum metric; generating inter-prediction samples for thetarget block using the bi-directional reference pictures and the refinedmotion vector generating residual samples based on the target block andthe inter-prediction samples; and encoding the residual samples, whereinthe first high level syntax element is decoded from a header of asequence.